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•Tlic present study cxajiuaeQ' three different methods of data collection 
in which subjects judged proxiinity between object pairs. One nietlK)d required 
subjects to partition objects into homogeneous subsets; the second entailed 
rating -^l>Jccl pairb, on a b imil-arity-dissimilarity continuum; and the thii^ 

'involved comparing inter-object proximities to a fixed standard. Tlie three 
types of proximities were analyzed by the noninetric multidimensional sc?iling, 
procedure, and subsequent multidimensional representations were compared for 
accuracy to a criterion or **truel' muLtidiinensional configuration of the s^me 

/obj^ects generated by thg,^same subjects. Considerable differences in accuracy 



were' found among tho m^hbdsr- 
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Nonnietric Multidiinensional Scaling: 
An Evaluation of Three^ata Collection Methods 
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Backa^oand 

Given a measure of proximity^ (similarity or dissimilarit)^ between each 
-pair, of n objects 0^,02,. ,o^^, nonmetric multidinjensionsl scaling (Kruskal, 
1964a, 1964lLuShepard, 1962a, 1962b) represents the obiects as Poirts ip t- 
dimensional space so that distances between pairs of points are monotonically 
^related tof^ corresponding input proximities. The proximities need only be ' 
distance-like (Shepar(^1972 , p. '^24) and measured ou at least an ^ordinal 
scale', thus methods of generating such'^'data are quite varied. 

— Due to the definitional generality of "proximity'^ cm 

exhaustive listing of data suitable for nonmetric multidimensional scaling 

■» 

is all but -impossible. However Cobinbs (1964), Torgerson (1958) and Wish 

- ■ ' ' y ' ■ 
(1972) have compiled taxonomies of commonly used proximity measures and ' ' 

methods for collecting them. . ■ " . . ' 

Since behavioral sciences focus on the activity of. animate entities, 
proximities are commonly generated by having 'subjects, cnroloy their cognitive, 
affect 6r psychomotor facilities in judging, inter-objedt similarities and 
dissimilarities. The present study deals with three such judgmental methods . 

' ^ t 

-Non- judgmental techniques constitute -a second source of proximity measures. 



Tliis class includes raethods ill which object i\ely measured properties of ob- 
'jects are used to derive a proximity index^ e.g. two objectS/Can be measured 
"on height, weight and other' relevajit attributes ^and the correlation or 'sum 
of squared,'Standardized differences acros,s these variables - taken as 
the inter-object proximity. Non- judgmental indices are also employed within 
the behavioral sciences and are particularly common to outside disciplines. 
The present study does not involve non- judgmental proximities. ^ ^ 

Inter-objdct proximities were judged subjects in the present Study 
in the following ways: (1) proximity grouping, (?) \proxijnitt rating and (3) 
proximity comparison. "^Each of these methods is des?)ribed in detail below. 



?K(:xvr\itij QWupaiQ. The instructions in proximity grouping tasks ask subjects 

*i ^ ' \ 

to sort n objects in^ mutually exclusive ajid exhaustive groups so t;hat ob- 

, !jccts in the same group aic luore siiiiilar tu each uthtjr than lo objects in 

/ , ' \ 

other groups* Tlie proportion (or frequei^cy) of tim.es objects o^ P j 
sorted into the same group is often used as a measure of proximity. 'Hiis 
type of index Has been employed iii multidimensional analysis of personalty 
traits (Rosenberg, Nelson 5 VivekaJian,than, 1968), nations (Wish, Deutsc^ 5 
Biner, 1970) aiid university faculty (Subkoviak 5 Levin, 1974). 

The judgment required in proximity grouping is .quite simple, and thui 
the method is particularly appropriate for use' with complex objebt propert\.es 
or'^^sopljistifcated subjects. Another advantage of the procedure is that si 
jects can respond to a large number of objects in a relatively shoi* span oJ 

* time.^ ' , • , — - 

* 

A^distinct disadvantage- of this approach is that a single sorting pro- 
vides n^ information aboUt proximity differences between objects within the. 

saiT^/group or about proxinity differences between groups. 'Tliese'^shortcomin^'s ca 
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be remedied, but at the expense of tijue tmd simplicity. For example, after 
the initial grouping, subjects caii be asked to judge proximities between dif- 
ferent groups or between objects wi thin ^ the same group. ^ 

V^ioxXmLtij Mating. Subjects are presented with all possible object pairs 

(o^,Oj) and are asked, to rate the proximity of each pair on a scale,. e.g. 

• "SIMILAR :0123456789: DISSIMILAl^." 'Hie average scale value of ' ' • 

pair (o-,o.) across all ratings is often used as a proximity measure. This 
> * 1 1 I . 

type of index has been employed in multidimensional analysis of^geometric . 
figures (Attneave, 1930), attitudes (Me'ssick, 1954) and interpersonal rela- 
tions (Wish, Kalplan § DeutsCh, 1974). 

A numbet of^ variations in mode of object presentation and type of rating 
.scale are possible (Torgerson, 1958;. Wish, 1972); but- the judgmental task re- ' 
mains basically one of .judging the absolute proximity of- each object p.air, 
as opposed to judging the proximities Ok£ various pairs relative to one another 
(see tlie proximity comp^arison method discussed below)*. > • ^ ' 

Complete information is obviously obtained about all n(n-l)/2 inter-object 



proximities, at the- expense of time and subject fatigue as n be come sT large. 
More important, the process of averaging ratings across subjects assumes that 

■i. 

the scale has the same meaning for^^lL subjects, this is typically a somewhat 
over-optimistic supposition in the light of subjects'" variable response^ ten- 
dencie's, e.g. attraction to or avoidance of the extreme ends of a rating 
locale. 

P/iOXAjfUXtj ccmpoJiUon.^' Subjects are asked to judge the degree of proximit)^^ 
betwveen object pairs (o^,Oj), relative to a standard. For exanple, subjects 
might report the percent of similarity between (o.,oO compared to the similar- 
.ity between a standard object-pair, and the average percent for (o.,o.) across 
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all subjects lakcn as a proximity rauasuic. TUis approach, has been used with 
much success in judging the geograpluc proximity o£ various world cities 
(Lundberg ^ Hbiaji, 1973}', and the moirhod {^eneraUzes easily to other stimuli-* 
^ Otlier variations of tlie comparative- jiWgment ap][)foaGh, wliich are not' ^ 
considered in the present study, are possible. For example, subjects might • 
be asked ,t,o provide a complete or weak (t^ies allowed) ranking of pairs (o^,Oj;) 
from most sijiiilar.pair to least similar pair, and the average rank of (0^,0^) * 
, across subjects taken as a proximity measure (Rapoport*,^ Fillenbaum,. 1972). 
However, as the number. n and hbmogeniety of objects increase, this task ' becomes 
quite difficult. Torgers^n (1958) and Coombs (1964) also deseribo indirect 
proeedures for trails forming object, comparisons to proximity data. . 

Like rating tecliniques, comparison m.ethgds afoduee complete information 

i ■ ' 

about inter- object* proximities for a greater investment of time. In addition, 
such procedqres may produce more valid and consistent data since -comparisons , 
among. actual 'objects tend to be better-defined than ratings on a somewhat 

^ambiguous scale, . ^ . . 

s . 

• . » 

, . UoXhod ' ■ ' 

' N. - ^ V • ^ 

SabjCcU. A total of 600 undergraduate and graduate students enrolled in 

communication arts and educat ional .psychology courses during the Spring Sem- 

ester of 1974 at the. University of Wisconsin' took part in the study. 

V 

f\cutCAl(iU> : * Tlie present study compared the accuracy of the aforementioned 
sorting; rating, and comparing methods iit judging distances (geographic pro- 
ximities^ between U.S. cities tobjects). . T^p sets of ten cities were cons id - 
ered: '(-1) Set 1 = {Philadelphia, Baltimore, Detroit, Atlanta, Qiicago, New ^ ^ 



Orleans,. Denver, Phoenix, Seattle, Los Mgcles) and (2) Set 2 = {Detroit-^in- 
cinnati, Atlanta, Minneapolis, St-. Louis, Kiuisas City, New Orleans, Denver, 
Phoenix, Houston}. The 10(10-l)/2 r 45 inter-city distances of Set 1 are 
heterogeneous (standard deviation 6(^0 miles) wliilc those of Set 2 are homo- 
goneous (standard deviation 365 mile^)". ;Thus inter-city distimces of Set 1 
are generally easier to differentiate than those of Set 2, and consequently 
judged proximities for Set 1 tend to be more consistent with aiv individual '3 
cognitii!c<5 map than for Set 2. ' , » / 

Qiiestionnaires were constructed for each of the tliree data collection 
methods, one involving the cities of' Set 1 (shoun below) and the second in- 
volving those of Set 2 W}ilcli were listed at the top of the questionnaire. 



The directions for the proximity groupi/g procedure wefe as follows. 

^ - 



' *'Scrrt the cities into separate .groups in tlie blank space^ 
bdc;;, so y^at cij:io3 in the saine group have small Jislajiccs ^ ^ 
betweei> tncm and are near one another. - Please sort each city 
into one' and only one group . Draw a circle around each sepa- 
rate group.' Use as f|w or as many groups ^as you think are *^ 
necessary; each group inay contain as few or as»many cities as 
seeirt appropriate^. ^ , ^ 

The instructions 'for the proxiiuity rating method were as fallows. 

' . -In Item 1 below rate the distance between Detroit and Los 
Angeles on the scale 0,1,2,3,4,5,6,7,8,9. Small nymbers 0-4 
indicate small distances, and the smaller the number the smaller 
the distance between the two cities. Large numbers 5-9 indicate 
larg-e distances, and the larger the number, the larger the dis- , r 
tance between the two cities.* . ^ * , ; 

Hepse circle one and only one numbei" 0,1,2,3,4,5,6,7,8,9 
in each of ttie following items. 

The 45 possible -pairs of cities'wcre then listed in random order, the order 

within each' pair also being raj:idomized. 
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T\\e directions for the prcximity compcrrison teclmique v;ere as follows. 

Imagine lluit the distance between New York City and San 
Francisco equals 100 units. Now compare the distcinces between 
the cities above to the distance between New York City and SaJi 
Francisco. For example, the distance between Detroit and LoS' 
Angeles is what percent 'of the distance between New York City 
and San Francisco? Record, your answer in Item 1 below. 

Complete aU the other items in the same way. Compare the 
distance between the given cities to the c^stance bet\^en New 
York City and San Francisco, and then record your answer as a 
percent. • - " ' ^ 



1 

Lii\ listec 



The 4 5* pairs oj:, cities were agaii\ listed in the same randomized order as for 
tlie proximity rating method. • , . 

After a-11 proximity grouping, rating or comparing judgments were made, 
subjects w^e^-e instructed to break the seal on the last page of the question- 
naire and to make a copy of their cognitive^ map on a completely blank outline 
representation of the 'Continental United SIaI-qs .(no man-made'^ or natural fea- 

,tures of^ajiy kind were depicted). Tlie inter-city distance^on this '^true** 
map \^rie used to determine the accuracy of' the grouping, rating or comparing 
judgm'ents of those same distances. The actual instructions were as follows. 

• • . ' • ■ ■ ■ ■ • 

" Use your mental picture or image of the U.S. to locate 
the following cities on the map provided below. 



1. 


Los Angeles 


6. 


Chicago 


2. 


- rtioenix 


^ 7. 


Denver ' 


-3. 


Baltimore 


■ 8.- 


New Orleans 


4. 


.Detroit 


9. 


Atlanta 


' 5. 


Seattle 


10. 


Philadelphia 



Use a dot (•) to indi^cate the loca^on of each city on^ 
the map below. Then miie the number of each city (1 thru 10 
above) over its dot (•). ' Plerfse be sure to place all 10 cities 
on till map. • / . • ' 

P/ioidduAt. ^The three data collectibn methods were completely crossed with 
the two sets of cities for a ^total of six conditions, and 100 subjects were ' 
randomly jfs-ligned to each conditiorL. Sifcjects were tested in.group^of jibout 



id 
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ZO-isO. The six fonns of ciucstioi-uKiirc were arranged in cyclical ordcT, fuid 
distributed one 'at a tinje, randomizing the assignment o£ subjects to coiTdi- 
l^ions (Under\N'00d, 1966, p; 115). Subjects then completed the, so;:ting, rating 
• or comparing judgments and the cognitive map/ The anbunt of time taken to 
complete the quest>ionnaires was recorded. 
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A nuj'nerical judgment S. . of'the proxijnity between the 45 possible pairs 
' of cities was obtained for each condition, small numbers indicating that a 
pair was geographically close and large numbers meaning the opposite. In 
the sorting t.ask a pair was'cocled 0 if a subject placed those two citj.es in * 
the saihfl group or 1 if they were placed in different groups ; in the rating 
task pairs were scored 0 thru, 9; and in the comparing task* pairs were gener- 
ally scored 0 thru 100 portent (a few pairs were judged greater than. 100 per- • 
? cent of the standard by a small number of subjects). Scores S^. for pai-r 

(o. ,0^) j.\'erc then averaged across the lOtf subjects in each condition to obtain 
a group proximity meaaurd S^. = ZS^^/lOO 'for ^the pair. ^ • \ 
The proximity measures S.. were next input into a nonmetric multidimen- 
sionai scaling program MINISSA-I (Lingoes, 1973), and a two 7 dimensional repre- 

sentation of the ten cities (defined by numerical coordinates) .was ^obtained 

' ^ • / ^ — 2 2" 

for-each condition. The 45 Euclidean distances d^^ = ^^^^il"* ^jl-^ "^.^^12 " "^jZ^ 

were computed between all pairs of cities in the representation, wlier^ • 

(x.-,,x.^) are ^the two-dimensional coordinates locating city 0. in the config- 

[ .uration for the 100 subjects. Tlie purpose of the study was to compare these 

• MJNISSA-I distances foy accuracy to the 45 corresponding true distances be- 

^ tw^,n cities orr .the cognitive maps of the same 100 subjects. 



- .Accordingly, the tine . coordinates C^^i>^^2^ locating each city o^ on 

each of the ..100 ^cognitive maps were obtained using an electronic, digitilzer 

that determines coordinates'^'ln units of 1^200^"^ of an inch. True Euclidean. 

distaiice^D- . = /fx., -Ti + (X., - X.,)^ were obtained for each subject . 
^ij V il J 1 1^ 3 ^ 

and averaged to provide .a group measure D^^ = ZD^^/100 of the true distance 
between pair (b^,Oj). 

Por puri)oses'of comparison *, the MINISSA-I distances d. . .wete transformed 

« ths ^ — 

to the^ same units*of measurement (1/200 of an inch) as true distances B^y 

— 2 

The new MINISvSA-I distances wdre given by dl^. = ^vhere a = Zd^^D^j/Zd^j 

— 2 J ' ^ 

is chosen to minimize Z(D. . - dl.) . This is an admisadble linear transfoma- 

tion of ratio scale distances (Lord f{ Novick, 1968, p. 21) and correspoaids to 

a unifom shrinking o^the MINISSA-I configuration to make it comparable to 
the' cognitive map. • ' - * 

/ -Finally, the t>'pical percent of discrepancy between true D. . arid ' 
MINISSA-I dl. across all 45 distaiices was computed as the measure of corre- *- 
spondence "between the actual cognitive maps and the. sorting, rating or compar- 
iing judgments of the 100 subjects in^a given ^condition (Kruskal, 1964a, p. 15) 



I [D. ; - d! .1^ 

1 . . ■ : ■ 

Percent Error = / - -^^ ^ 

For example, as sho™ in Table 1 for sortings of Set 1 cities by N = 100 sub- 
jects; d^ typically differ from true D^. by 32 percentT 

Finally, to determine if results are consistent for various size groups,* 
the 100 subjects in each condition were randomly partitioned into five groups 
of N = 20 and independently into ten groups of N =^0 subjects; and the 
analysis described above was. repeated for each of the smaller groups. 

10 



Table 1 shows, tlie percent of dis'crepancy between true D'^j and MIl^ISSA-I- 
for variouj^' size groups N = 100,^0, 10 under the six method x stimuli 
conditions; for group size N = 20 and K = 10, the median percent error across 
five and ten subgroups resj^ectively is reported. A number of conclusions . 
can be drawn from these« result%. 



, ' Insert Table 1- about here 

^ 



First, for the heterogeneous stimuli of "Set .1 and for a given group -size 
N, sorting distances dl . containecl about four times more error than either • 
rating or comparing dist^ces. Thus the tendency of subjects to sort • . ^ 
Set 1 cities into the sine clusters and the resulting lack of proximity^ in- 
formation atoift cities Vithin a cluster introduced coi^iderable error into 
*the noninetric multidimensional scaling configuration. As one wouTd' expect , 
thiSg condition became more severe with" decreasing group size and ^ 
loss iof intra-cluster infomation due to the increased likelihood of com- 
plete agreement among subjects' sortings. Increased agreement among the 
ratings find compariaqis of smaller groups and. the slight reductic^ in discrim 

inations ajiiong dike proximities had a similar effect orf the configurational 

\ ' 

accuracy of these tasks. ^ 

"/ ' ' > • • 

^ Second, for the, homogeneous stimuli of Set 2, the sorting, rating and 

comparing methods .w^re equally accurate for applied purposes, except for 

small samples' Fl < 10; in fact, the sorting method produced noticeably more 

accfurate results for Set 2 than Set l\dufe to subjects' greater variability 

in defining Set 2 clusters. . The fornier result has interesting practical impl 

■ cations. In the beliavipVali sciences, a proximity S. . is nonnally perceived 

11 



variably acros§ subjects, as for Set 2 stimuli.. Onder such conditions, the 
sorting procedure tends to produce results as .accurate as rating or compariiig, 
assuming -N > 10. Purthenjiore, sorting requires considerably less tijne than 

. rating or comparing. For only n = 30 objects^ in* the present study ,^ the 
average sorting time was 7 1/2 minutes as opposed to 11 and ll 1/2 minutes 
for rating and comparing. Since sorting time is a fxinction of n while 
rating\a]id^ CQTnparing time are a .function of n(n-l)/2, this difference g^ows 

'^rapidly as n/increases. . . ' • ' 
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Table 1 , : / , . \ / ' 
Percent Error Between True Cognitive Map and Derived •// 
Nonme trie Multidimensional Scaling ^nfigurat ion . 
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Method 


Sorting - 




Rating 




.'13cai^aring 




Stimuli 


N = 100 


N = 20 


N =■ 10 


N = 100 


N = 20 


N = 10' 


N = lOO- 




:N-=, 10; 


Set 1 


.* .32 


.40 


-42 


.09 


,;12 - 


^.14 


.08 






Set 2 


.14 


.21 
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.17 




■.17 


.15 
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